Image processing apparatus and control method of image processing apparatus

ABSTRACT

An image processing apparatus generates one image by using at least one frame each of a plurality of moving images obtained by taking moving images of a plurality of different regions of an eye at different times. The apparatus includes a deciding unit configured to decide the at least one frame in each of the plurality of moving images, so that regions which have actually been shot are included in the plurality of moving images in the plurality of regions; and an image generating unit configured to generate one image by using the at least one frames decided from each of the plurality of moving images.

BACKGROUND OF THE INVENTION

Field of the Invention

The present invention relates to an image processing apparatus used inopthalmological diagnosis and treatment, and to a control method of theimage processing apparatus.

Description of the Related Art

Examination of the eye is widely performed for early diagnosis andtreatment of lifestyle diseases and diseases which are primary causes ofloss of eyesight. The scanning laser opthalmoscope (SLO), which is anopthalmological apparatus that employs the principle of confocal laserscanning microscopy, performs raster scanning of a laser, which ismeasurement light, over the fundus, and acquires a high-resolutionplanar image from the intensity of returning light at high speed. Anapparatus which images such a planar image will hereinafter be referredto as an SLO apparatus, and the planar image to as an SLO image.

In recent years, increased beam diameter of measurement light in SLOapparatuses has enabled acquisition of SLO images of the retina, withimproved horizontal resolution. However, the increased beam diameter ofthe measurement light has led to a problem of deterioration the S/Nratio and the resolution of the SLO image during acquisition of SLOimages of the retina, due to aberration of the eye being examined. Anadaptive optic SLO apparatus has been developed to solve this problem.The adaptive optic SLO apparatus has an adaptive optic system thatmeasures aberration of the eye being examined in real time using awavefront sensor, and corrects aberration occurring in the eye beingexamined with regard to the measurement light and the returning lightthereof using a wavefront correction device. This enables SLO imageswith high horizontal resolution (high-magnification image) to beacquired.

Such high-magnification images can be acquired as moving images, and areused for non-invasive observation of hemodynamic states. Retinal bloodvessels are extracted from each frame, and the movement speed of bloodcells through the capillaries, and so forth, are measured.Photoreceptors P are detected and the density distribution and array ofthe photoreceptors P measured, to evaluate the relationship with visualfunctions, using high-magnification images. FIG. 6B shows an example ofa high-magnification image. The photoreceptors P, a low-luminance regionQ corresponding to the position of capillaries, and a high-luminanceregion W corresponding to the position of a white blood cell, can beobserved.

In a case of observing photoreceptors P or measuring distribution ofphotoreceptors P using a high-magnification image, the focus position isset nearby the outer layer of the retina (B5 in FIG. 6A) to take ahigh-magnification image such as in FIG. 6B. On the other hand, thereare retinal blood vessels and capillaries that have branched runningthrough the inner layers of the retinal (B2 through B4 in FIG. 6B). Incases of taking photographs of the eye to be examined, image region tobe imaged may be larger than the angle of view high-magnification image.Cases of imaging widespread photoreceptor defect regions, cases ofimaging a parafovea region which is an area of predilection forearly-stage capillary lesions, and so forth, fall under such cases.Accordingly, Japanese Patent Laid-Open No. 2012-213513 discloses atechnology to composite and display multiple high-magnification imagesacquired by shooting at different shooting positions.

Also, Japanese Patent Laid-Open No. 2013-169309 discloses a technologyin which exceptional frames where effects of ocular microtremor in ahigh-magnification moving image of a certain shooting position aredetermined, and just the frames other than the exceptional framesdetermined in the high-magnification moving image are displayed.

SUMMARY OF THE INVENTION

An image processing apparatus generates one image by using at least oneframe each of a plurality of moving images obtained by taking movingimages of a plurality of different regions of an eye at different times.The apparatus includes a deciding unit configured to decide the at leastone frame in each of the plurality of moving images, so that regionswhich have actually been shot are included in the plurality of movingimages in the plurality of regions; and an image generating unitconfigured to generate one image by using the at least one framesdecided from each of the plurality of moving images.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments with reference to theattached drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a functional configurationexample of an image processing apparatus according to a first embodimentof the present invention.

FIGS. 2A through 2C are block diagrams illustrating configurationexamples of a system including the image processing apparatus accordingto an embodiment of the present invention.

FIG. 3 is a diagram for describing the overall configuration of an SLOimage imaging apparatus according to an embodiment of the presentinvention.

FIG. 4 is a block diagram illustrating a hardware configuration exampleof a computer which has hardware equivalent to a storage unit and imageprocessing unit and holds other units as software which is executed.

FIG. 5 is a flowchart of processing which the image processing apparatusaccording to an embodiment of the present invention executes.

FIGS. 6A through 6J are diagrams illustrating what is performed in imageprocessing according to the first embodiment of the present invention.

FIGS. 7A and 7B are flowcharts illustrating the details of processingexecuted in S530 and S540 according to the first embodiment of thepresent invention.

FIG. 8 is a block diagram illustrating a functional configurationexample of an image processing apparatus according to a secondembodiment of the present invention.

FIGS. 9A through 9E are diagrams illustrating what is performed in imageprocessing according to the second embodiment of the present invention.

FIGS. 10A and 10B are flowchart illustrating the details of processingexecuted in S530 and S540 according to the second embodiment of thepresent invention.

FIG. 11 is a diagram for describing the overall configuration of atomographic image imaging apparatus according to a third embodiment ofthe present invention.

FIGS. 12A through 12D are diagrams illustrating what is performed inimage processing according to the third embodiment of the presentinvention.

FIG. 13 is a block diagram illustrating a functional configurationexample of an image processing apparatus according to a fourthembodiment of the present invention.

FIGS. 14A through 14C are diagrams illustrating what is performed inimage processing according to another embodiment of the presentinvention.

DESCRIPTION OF THE EMBODIMENTS

Now, a case will be considered where a certain frame is selected fromeach of multiple high-magnification moving images acquired by shootingat different shooting positions, and the selected frames are composited(montaged). Several frames are generally selected from the frames of themultiple high-magnification moving images, and the selected frames areused to acquire a representative image. The acquired representativeimages are composited, thereby generating a wide-range image. There havebeen cases where the continuance in adjacent representative images wasnot good with regard to shooting position, luminance properties, imageproperties, and so forth, when adjacent representative images werecompared with each other. There were cases where unanalyzable regionsoccurred, regions for analysis could not be extracted, and so forth,when measuring the distribution of cell groups, tissues, and lesionsthereof (photoreceptor defects, microaneurysms) distributed over a widearea, using such wide-range images.

There has been found to be demand for selecting images when acquiringrepresentative images of each of multiple high-magnification movingimages acquired by shooting at different shooting positions, so that thecontinuity of the representative images is improved.

The image processing apparatus according to the present embodiment has aselecting unit (e.g., selecting unit 134 in FIG. 1) that selects imagesfrom each of multiple moving images that have been taken at differentpositions of the eye, based on continuity of properties of multipleimages (an image group) made up of images (representative images)acquired by selecting from each of the multiple moving images.Accordingly, when acquiring representative images from each of multiplehigh-magnification moving images shot at different shooting positions,images can be selected so that the continuity among the representativeimages is better.

Now, properties of multiple images (an image group) are, for example, atleast one of the relative position, luminance properties, and imageproperties, of the multiple images. Each image (image acquired by beingselected from a moving image) of the multiple images (image group) is arepresentative image acquired from the moving image, and may be oneimage selected from a moving image, or may be multiple images withrelatively little noise, artifacts, and so forth, which are selected andoverlaid. In a case of using overlaid images, the number of overlaidimages is preferably small, so that the continuity among the propertiesof the multiple images will be high. Also preferably further provided isa determining unit, to determine a value indicating continuity, wherebyan image can be selected from a moving image so that the determinedvalue satisfies predetermined conditions. A case where a determinedvalue satisfies predetermined conditions here is, for example, a casewhere the determined value exceeds a threshold value, or is greatest. Avalue indicating continuity is preferably determined using a compositedimage where multiple images have been composited. For example,determination is made based on at least one of the area of thecomposited image and the length of avascular area boundary, which willbe described in detail in the embodiments. Preferred embodiments of theimage processing apparatus according to the present invention and theoperation method thereof will be described below in detail withreference to the attached drawings. Note however, that that the presentinvention is not restricted to these.

Also, it is difficult to acquire images with no image distortion in anyof the images when consecutively shooting at multiple shootingpositions. Accordingly, generally, high-magnification moving imageshooting is performed of a certain shooting position of the eye. Now,Japanese Patent Laid-Open No. 2012-213513 discloses that shooting may beperformed multiple times at the same position, and that images used toconfigure a panorama image may be selected such that images with thebest correlation with adjacent images are selected. Now, there have beenproblems of incomplete images in the panorama image if just imagesoptimal to configure a panorama image are selected from multiple images.Accordingly, an image processing apparatus according to anotherembodiment selects multiple frames from moving images, so thatincomplete images in the panorama image are minimized. Accordingly,incomplete images in the panorama image can be reduced.

Preferred embodiments of the image processing apparatus according to thepresent invention and the operation method thereof will be describedbelow in detail with reference to the attached drawings. Note however,that that the present invention is not restricted to these.

First Embodiment Continuity of Relative Position or Luminance Propertiesof Multiple Images of Different Positions

The image processing apparatus according to a first embodimentdetermines suitability of an image group based on continuity of at leastone of relative position and luminance properties of multiple images (animage group) at different positions. This configuration has been made sothat a region of photography can be observed under generally the sameconditions, by selecting, compositing, and displaying frames or imageswith the highest suitability.

Specifically, description will be made regarding a case where an imagegroup is made up of nine high-magnification images such as illustratedin FIG. 6G, an overlaid image is generated using frames selected by aselecting unit in increments of the shooting positions, the overlaidimages are composited, and the suitability as an image group isdetermined.

Overall Configuration

FIG. 2A is a configuration diagram of a system including the imageprocessing apparatus 10 according to the present embodiment. The imageprocessing apparatus 10 is connected to an SLO image imaging apparatus20 and data server 40 via a local area network (LAN) 30 includingoptical fiber, Universal Serial Bus (USB), IEEE 1394, or the like, asillustrated in FIG. 2. The configuration of connection to these devicesmay be via an external network such as the Internet, or may be aconfiguration where the image processing apparatus 10 is directlyconnected to the SLO image imaging apparatus 20.

First, the SLO image imaging apparatus 20 is an apparatus to shootwide-angle images D1 and high-magnification images Dh of the eye. TheSLO image imaging apparatus 20 transmits wide-angle images D1,high-magnification images Dh, and information of fixation targetpositions F1 and Fh used for shooting thereof, to the image processingapparatus 10 and the data server 40. In a case where images at eachmagnification are acquired at different shooting positions, this isexpressed as D1I, Dhj. That is to say, i and j are variables indicatingthe numbers for the shooting positions, where i=1, 2, . . . , imax, andj=1, 2, . . . , jmax. In a case of acquiring high-magnification imagesat different magnifications, this is expressed like D1 j, D2 k, . . . ,in order from the highest-magnification image, with D1 j denoting ahigh-magnification image and D2 k denoting a mid-magnification image.

The data server 40 holds the wide-angle images D1 and high-magnificationimages Dh of the eye, shooting conditions data such as fixation targetpositions F1 and Fh used for the shooting thereof, image features of theeye, normal values relating to distribution of the image features of theeye, and so forth. In the present invention, image features relating tothe photoreceptors P, capillaries Q, blood cells W, retinal bloodvessels, and retina layer boundary, are handled as image features of theeye. The wide-angle images D1, high-magnification images Dh output fromthe SLO image imaging apparatus 20, fixation target positions F1 and Fhused for the shooting thereof, and image features of the eye output fromthe image processing apparatus 10, are saved in the server. Also, thewide-angle images D1, high-magnification images Dh, image features ofthe eye, and normal value DATA of image features of the eye, aretransmitted to the image processing apparatus 10 in response to requestsfrom the image processing apparatus 10.

Next, the functional configuration of the image processing apparatus 10according to the present embodiment will be described with reference toFIG. 1. FIG. 1 is a block diagram illustrating the functionalconfiguration of the image processing apparatus 10. The image processingapparatus 10 includes a data acquiring unit 110, a storage unit 120, animage processing unit 130, and an instruction acquiring unit 140. Thedata acquiring unit 110 includes an image acquiring unit 111. The imageprocessing unit 130 includes a positioning unit 131, an individual imagedetermining unit 132, an image group determining unit 133, the selectingunit 134, and a display control unit 135. Further, the image groupdetermining unit 133 includes a position determining unit 1331 and aluminance determining unit 1332.

SLO Image Imaging Apparatus that has Adaptive Optic System

Next, the configuration of the SLO image imaging apparatus 20 that hasan adaptive optic system will be described with reference to FIG. 3.First, 201 denotes a light source, for which a super luminescent diode(SLD) light source was used. Although the light source is shared betweenfundus imaging and wavefront measurement in the present embodiment, aconfiguration may be made where these are separate light sources thatare multiplexed along the way. Light irradiated from the light source201 passes through a single-mode optic fiber 202 and is irradiated asparallel measurement light 205 from a collimator 203. The irradiatedmeasurement light 205 is transmitted through a light splitting unit 204made up of a beam splitter, and is guided to an optical system of theadaptive optic system.

The adaptive optic system is made up of a light splitting unit 206, awavefront sensor 215, a wavefront correcting device 208, and reflectingmirrors 207-1 through 207-4 for guiding light thereto. The reflectingmirrors 207-1 through 207-4 are installed so that at least the pupil ofthe eye, the wavefront sensor 215, and the wavefront correcting device208 are optically in a conjugate relationship. A beam splitter is usedin the present embodiment as the light splitting unit 206. Also, aspatial phase modulator using a liquid crystal device is employed as thewavefront correcting device 208 in the present embodiment. Note that avariable shape mirror may be used as the wavefront correcting device.Light that has passed through the adaptive optic system is scannedone-dimensionally or two-dimensionally by a scanning optical system 209.As the scanning optical system 209, two Galvano scanners were used inthe present embodiment of main scanning (fundus horizontal direction)and sub-scanning (fundus vertical direction). A resonance scanner may beused for the main scanning side of the scanning optical system 209 forhigher speed shooting. An eye 211 is scanned by the measurement light205 scanned by the scanning optical system 209, via eyepiece lenses210-1 and 210-2. The measurement light 205 by which the eye 211 isirradiated is reflected or scattered at the fundus. Optimal irradiationfor the eyepiece visibility of the eye 211 can be realized by adjustingthe position of the eyepiece lenses 210-1 and 210-2. Although eyepiecelenses are used here, sphere mirrors or the like may be used for theconfiguration.

Part of the reflected/scattered light (returning light) reflected fromor scattered at the retina of the eye 211 travels along the same path asit came when input but in the opposite direction, and is reflected atthe wavefront sensor 215 by the light splitting unit 206 to be used tomeasure the beam wavefront. The wavefront sensor 215 is also connectedto an adaptive optic control unit 216, and informs the adaptive opticcontrol unit 216 of the received wavefront. The wavefront correctingdevice 208 also is connected to the adaptive optic control unit 216, andperforms modulation instructed by the adaptive optic control unit 216.The adaptive optic control unit 216 calculates a modulation amount(correction amount) for correction to a wavefront without aberration,based on the wavefront acquired from the measurement results by thewavefront sensor 215, and instructs the wavefront correcting device 208to perform such modulation. Note that wavefront measurement andinstruction to the wavefront correcting device 208 is performedrepeatedly, with feedback control being performed to constantly have anoptimal wavefront.

Part of the reflected/scattered light that has been transmitted throughthe light splitting unit 206 is reflected by the light splitting unit204, passes through a collimator 212 and optic fiber 213 and is guidedto a light intensity sensor 214. The light is converted into electricsignals at the light intensity sensor 214, formed into an image servingas an eye image by a control unit 217, and displayed on a display 218.By increasing the oscillation angle of the scanning optical system inthe configuration illustrated in FIG. 3 and the adaptive optic controlunit 216 instructing not to perform aberration correction, the SLO imageimaging apparatus 20 can also operate as a normal SLO apparatus, and cantake wide-angle SLO images (wide-angle image D1).

Hardware Configuration and Execution Procedures of Image ProcessingApparatus 10

Next, the hardware configuration of the image processing apparatus 10will be described with reference to FIG. 4. In FIG. 4, 301 denotes acentral processing unit (CPU), 302 memory (random access memory (RAM)),303 control memory (read-only memory (ROM)), 304 an external storagedevice, 305 a monitor, 306 a keyboard, 307 a mouse, and 308 aninterface. Control programs for realizing the image processing functionsaccording to the present embodiment, and data used at the time of thecontrol programs being executed, are stored in the external storagedevice 304. The control programs and data are loaded to the RAM 302 viaa bus 309 as appropriate under control of the CPU 301, executed by theCPU 301, and function as the units described below. The functions of theblocks making up the image processing apparatus 10 will be correlatedwith specific execution procedures of the image processing apparatus 10illustrated in the flowchart in FIG. 5.

Step 510: Image Acquisition

The image acquiring unit 111 requests the SLO image imaging apparatus 20to acquire wide-angle images D1, high-magnification images Dhj, andcorresponding fixation target positions F1 and Fh. In the presentembodiment, the fixation target positions F1 and Fh are set at the foveaof the macula, and wide-angle images D1 and high-magnification imagesDhj are acquired. Note that the setting method for shooting positions isnot restricted to this, and may be set to any position.

The SLO image imaging apparatus 20 acquires and transmits the wide-angleimages D1 and high-magnification images Dhj, and the correspondingfixation target positions F1 and Fh, in accordance with the acquisitionrequest. The image acquiring unit 111 receives the wide-angle images D1,high-magnification images Dhj, and fixation target positions F1 and Fh,from the SLO image imaging apparatus 20 via the LAN 30, and stores thesein the storage unit 120. Note that the wide-angle image D1 andhigh-magnification image Dhj in the present embodiment are moving imagesof which the inter-frame positioning has already been performed.

Step 520: Positioning

The positioning unit 131 performs positioning of the wide-angle imagesD1 and the high-magnification images Dhj, and obtains the relativeposition of the high-magnification images Dhj upon the wide-angle imageD1. In a case where there is an overlapping region among thehigh-magnification images Dhj, the inter-image similarity is calculatedregarding this overlapping region as well, and positions thehigh-magnification images Dhj with each other at the position where theinter-image similarity is the greatest.

Next, in a case where images of different magnification have beenacquired in S510, positioning is performed from lower magnificationimages. For example, in a case where a high-magnification image D1 j anda mid-magnification image D2 k have been acquired, first, positioning isperformed between the wide-angle image D1 and the mid-magnificationimage D2 k, and next position is performed between the mid-magnificationimage D2 k and the high-magnification image D1 j. In a case where thereare only high-magnification images, it is needless to say thatpositioning is performed only between the wide-angle image D1 and thehigh-magnification image D1 j.

Note that the positioning unit 131 acquires the fixation target positionFh used for shooting the high-magnification image Dhj from the storageunit 120, and uses this to set a search start point for a positioningparameter in the positioning between the wide-angle image D1 and thehigh-magnification image Dhj. Any known technique can be used forinter-image similarity or coordinate conversion techniques. In thepresent embodiment, a correlation coefficient is used for inter-imagesimilarity, and Affine transform is used as the coordinate conversiontechnique to perform positioning.

Step 530: Processing to Determine Suitability of Each Moving Image

The individual image determining unit 132 performs determinationprocessing for suitability, based on the luminance values of the framesand the inter-frame movement amount. Further, the selecting unit 134performs selection processing based on the suitability determinationresults, and forms individual images. An individual image here may be animage where all frames of the moving image have been overlaid, or may beone selected frame. An image where multiple images with relatively smallnoise and so forth are selected and the selected images are overlaid maybe used. The processing of this step will be described in detail laterwith reference to the flowchart in FIG. 7A.

Step 540: Processing to Determine Suitability as Image Group

Based on the individual images formed in S530, the image groupdetermining unit 133 determines suitability of an image group (multipleimages of adjacent different positions) based on relative position andrelative luminance therebetween, the selecting unit 134 selects acomposition of images with the highest suitability, and composites theseto form an image. In the present embodiment, the selecting unit 134selects a frame interval of the composition regarding which the imagegroup determining unit 133 has determined that the suitability is thehighest at each acquiring position, performs overlaying to generate animage, and forms a composited image. The processing of this step will bedescribed in detail later with reference to the flowchart in FIG. 7B.

Step 550: Display

The display control unit 135 displays the high-magnification images Dhjupon the wide-angle image D1, based on the value of a positioningparameter obtained in S520, or on the region, frame, or image selectedin S540. The display control unit 135 may correct difference inconcentration among high-magnification images for display, in a casewhere multiple high-magnification images Dhj have been acquired. Anyknown luminance correction method may be used. In the presentembodiment, a histogram Hj is generated for each high-magnificationimage Dhj, and linear transform of the luminance values of thehigh-magnification images Dhj is performed so that the average andvariance of the histograms Hj are common values among thehigh-magnification images Dhj, thereby correcting the difference inconcentration. Note that the luminance correction method amonghigh-magnification images is not restricted to this, and any knownluminance correction method may be used. Further, with regard to thedisplay magnification, a high-magnification image which the operator hasspecified via the instruction acquiring unit 140 is enlarged anddisplayed on the monitor 305.

Step 560: Instruction of Whether or not to Save Results

The instruction acquiring unit 140 externally acquires an instructionregarding whether or not to save to the data server 40 the wide-angleimages D1, the high-magnification images Dhj selected by the selectingunit 134, the fixation target positions F1 and Fh, and the positioningparameter values acquired in S520. This instruction is input by theoperator from the keyboard 306 or mouse 307, for example. In a casewhere saving has been instructed, the flow advances to S570, and in acase where saving has not been instructed, the flow advances to S580.

Step 570: Saving Results

The image processing unit 130 correlates the date and time of theexamination, information identifying the examined eye, the wide-angleimages D1, the high-magnification images Dhj selected by the selectingunit 134 and the fixation target positions F1 and Fh, and thepositioning parameter values, and transmits to the data server 40.

Step 580: Instruction of Whether or not to End Processing

The instruction acquiring unit 140 externally acquires an instructionregarding whether or not to end processing by the image processingapparatus 10 regarding the wide-angle images D1 and high-magnificationimages Dhj. This instruction is input by the operator from the keyboard306 or mouse 307, for example. In a case where an instruction to endprocessing has been acquired, the processing ends. On the other hand, ina case where an instruction to continue the processing has beenacquired, the flow returns to S510, and processing is performed on thenext eye to be examined (or processing is performed again regarding thesame eye for examination).

Processing Regarding Determination of Suitability for Each Moving Image

Next, processing executed in S530 will be described in detail withreference to the flowchart in FIG. 7A.

Step 710: Acquisition of Suitability Determination Standard

The individual image determining unit 132 acquires a suitabilitydetermination standard via the instruction acquiring unit 140. Thefollowing items a) through d) are listed here as suitabilitydetermination standards;

a) that the luminance value of the image is in an appropriate range,

b) range of appropriate values of image quality (S/N ratio, etc.),

c) that the amount of movement as to the reference frame is in anappropriate range, and

d) that the focus position is in an appropriate range, where, a) isacquired as the suitability in the present embodiment. This is toexclude low-luminance frames which occur due to measurement light notreaching the fundus, as a result of blinking or marked deviation offixation position.

Step 720: Suitability Determination

The individual image determining unit 132 determines suitability foreach frame of the high-magnification SLO image Dhj, following thestandard acquired in S710. In the present embodiment a value of 1 isassigned if each a) is in an appropriate range, and a value of −1 isassigned if out of an appropriate range.

Step 730: Image Selection

The selecting unit 134 selects images (frames in a case of movingimages) to use for display, in increments of shooting positions, basedon the suitability determined in S720, and forms an image. In thepresent embodiment, a high-magnification image is a moving image wherephotoreceptors have been imaged as illustrated in FIG. 6C, and anoverlaid image is formed from the moving image. The following items (i)and (ii) can be listed as principles for forming individual images here,which is to

(i) maximize the number overlaid (priority given to image quality), and

(ii) maximize area of overlaid image (priority given to prevention ofincomplete images).

In the case of (i), all frames selected in S720 are used for overlaying.For example, in a case where the position of each frame in an individualhigh-magnification moving image is correlated (Nf: frame No.) asillustrated in FIG. 6(c), the results of overlaying are as illustratedin FIG. 6D. In this example, the leading frame is the reference frame.Regions not used for overlaying (incomplete images) are indicated byblack in FIG. 6D. While high quality images, regions with frames havingno pixel values as a result of inter-frame positioning are not used inthe overlaying, so incomplete images readily occur. In the case of (ii),frames selected in S720 which have even slight positional deviation areexcluded. For example, in the case of FIG. 6C, frames Nos. 2 through 4are excluded. While incomplete images do not occur, the number of imagesoverlaid is smaller, so the image quality tends to be lower than in thecase of (i). Now, in the case of the present embodiment, frames withsuitability of 1 calculated in S720 are selected, and an overlaid imageis formed following the principle (i), which is to say that only regionswhere the pixel values are positive in all selected frames are used.

Processing to Determine Suitability as Image Group

Next, the processing executed in S540 will be described in detail withreference to the flowchart in FIG. 7B.

Step 740: Suitability Determination

The image group determining unit 133 composites the image group formedin S730 following the positioning parameter used in S520, and determinessuitability of the image group based on the relative position andluminance properties of the image group. Assumption will be made in thepresent embodiment that the image group is made up of nine overlaidimages such as illustrated in FIG. 6G, and that the image No. jincreases in raster scanning (zigzag scanning) order from the upperleft. Determination principles relating to suitability of the compositedimages (image group) are as listed below in order of priority, which isto say,

1. that no incomplete-image region is generated in the composited image,

2. that the image quality does not vary according to the shootingposition, and

3. that as many images as possible are overlaid.

Of these, 1 and 2 are conditions set for enabling within the compositedimage to be observed under the same conditions, with the positiondetermining unit 1331 determining condition 1, and the luminancedetermining unit 1332 determining condition 2. Both of the relativeposition and luminance properties of the image group do not have to beset as conditions; setting either one as a condition is sufficient.

Now, in a case where suitability as an image group is not determined,i.e., in a case where the composited image is generated following theprinciple of (ii) in S730 for each shooting position, there will becases where the above condition 2 is not satisfied. Also, in a casewhere the composited image is generated following the principle of (ii)such that the above condition 2 is satisfied, the image quality is lowerthan a case where suitability of the image group is determined andimages are selected, since the number of overlaid images has to matchthat of the image with the fewest overlaid images. Accordingly,performing suitability determination taking into consideration datacontinuity and complementation at the edges and overlapping portions ofadjacent images enables a higher-quality composited image to be obtained(overlaying being performed with a greater number of images), whilesatisfying conditions 1 and 2.

In the present embodiment, there are redundant regions between twoadjacent images, such as indicated by the gray regions in FIG. 6F, andredundant regions between four adjacent images, such as indicated by theblack regions in FIG. 6F. Specifically, the suitability of the imagegroup is determined by the following procedures.

(1) Individual images generated in S730 (priority on image quality) arecomposited according to the positioning parameters obtained in S520.

(2) Whether or not there are incomplete images within the compositedimage generated in (1) is checked, and(area of composited image−area of incomplete images)/(area of compositedimage)is calculated.Step 750: Image Selection

The selecting unit 134 performs image selection in eachhigh-magnification image so that the suitability is the highest, basedon the suitability determined in S740, and performs formation processingof the image group based on the selected images. Specifically, imageselection (selection of frame or region) is performed according to thefollowing procedures, and image group formation processing is performed.

(3) If there are no incomplete images, the composited image is formed asit is and the processing ends.

(4) If there are incomplete images, the position of the incomplete-imageregion is obtained.

(5) Check whether or not there is complementary (substitute) data in aredundant region of an image including the incomplete-image region orhaving a side adjacent to the incomplete-image region. For example, inFIG. 6G there are incomplete images in image 6 and image 9, so whetheror not there is complementary data at image 6 and the left edge of image5, and in image 9, is checked.

(6) If there is complementary (substitute) data, the incomplete imageregion is replaced with complementary data that has the best imagequality (the number of overlaid images is the greatest) of thecomplementary data, and (8) is executed (equivalent to region selectionprocessing in the image by the selecting unit 134).

(7) If there is no complementary (substitute) data, selected frames ofimages having images including incomplete-image regions or a sideadjacent to the region are changed so that the incomplete-image regionis resolved. If there are multiple frame selection methods to resolvethe incomplete-image region, the frame selection method where the numberof overlaid images is the greatest is selected.

(8) The number of overlaid images ANmin that is the smallest number ofoverlaid images out of the overlaid image group obtained in (7) is setas the number of overlaid images of the composited image, the number ofoverlaid images at each shooting position is changed to ANmin, and theoverlaid image is generated again.

(9) The overlaid image generated in (8) is used to generate a compositedimage. There are no more incomplete images, as illustrated in FIG. 6H,and a composited image where the numbers of overlaid images are the sameand are maximal is generated.

Note that in a case where moving images have been acquired Nt times(Nt≧2) at the same shooting position in the same examination, the movingimage with the highest suitability (of the first through Nt'th times) isselected in S730, whether or not there is substitute data regarding theincomplete-image region in a different shooting time is checked in imageselection (5) in S750, and substitution is made with the substitutiondata having the highest image quality of all substitution data. If therestill is an incomplete-image region remaining, whether or not there issubstitution data in a redundant region of an adjacent image may bechecked.

Note that while the composited image formed based on the determinationof suitability as an image group has been described as being a stillimage (overlaid image) in the present embodiment, the present inventionis not restricted to this. For example, suitability determination may beperformed taking into consideration complementation of data at the edgesand overlapping portions of adjacent moving images, and a moving imagemay be composited and displayed thereupon, as illustrated in FIG. 6J.

The flow of the basic processing in a case of composited display ofmoving images is the same as in a case of composited display of stillimages, but the following points differ as follows.

(i) A time phase data acquiring apparatus 50 such as illustrated in FIG.2B is connected to the image processing apparatus 10, and time phasedata is acquired simultaneously with the moving image. Time phase datais biosignal data acquired by a sphygmograph, for example. Referencingthe time phase data yields the cardiac cycle of each moving image, i.e.,which cycle should be played. The playback cycle is aligned to be thesame among the moving images by frame interpolation processing of themoving images.

(ii) The longest continuous frame section from which frames withabnormal luminesce have been removed is selected in the image formationprocessing in increments of shooting position in S730.

(iii) Suitability determination is performed in the suitabilitydetermination processing as an image group in S740 according to thefollowing principles, which is to say that

-   -   no incomplete-image regions generated in composited image,    -   no variance in number of playback frames from one shooting        position to another, and    -   composite and display moving images of as many frames (cycles)        as possible.

(iv) The frames selected in (6) (7) (8) of the image formationprocessing as an image group in S750 are set as a continuous framesection.

Accordingly, incomplete images are eliminated from the compositeddisplay of the moving image, and a composited moving image with thelongest continuation of frames having the same number of playback framesis formed. In a case where no time phase data is acquired, a compositeddisplay may be made as a moving image without adjusting the playbackclock time.

According to the configuration described above, when displaying acomposited image of adaptive optic SLO images at different shootingpositions, the image processing apparatus 10 determines suitability ofan image group in a case of comparing with a region to be shot, i.e.,determines suitability based on how small an unobservable region is.Regions or frames or images are selected from images based on datacontinuity and complementation at edges or overlapping regions ofadjacent images, so that the suitability is greatest, and composited anddisplayed. Accordingly, in a case where cells and tissue to be observed,and lesions thereof, exist across multiple high-magnification images, acomposited image which can be observed under generally the sameconditions can be generated.

Second Embodiment Continuity of Image Features of Multiple Images atDifferent Positions

An image processing apparatus according to a second embodiment isconfigured to determining suitability of an image group based on thecontinuity of image features extracted from adjacent high-magnificationimages, rather than determining suitability of an image group based oncontinuity of relative position and luminance properties of adjacenthigh-magnification images as in the first embodiment. Specifically, thesuitability of an image group is determined based on the continuity ofcapillary regions of the parafovea extracted from high-magnification SLOimages.

The configuration of apparatuses connected with the image processingapparatus 10 according to the present embodiment is the same as in thefirst embodiment. The data server 40 holds, besides the wide-angleimages D1 and high-magnification images Dh of the inspected eye, andacquisition conditions such as fixation target positions F1 and Fh usedfor the acquisition thereof, image features of the eye and normal valuesrelating to distribution of the image features of the eye. While anyimage features of the eye can be held, image features relating to theretinal blood vessels, capillaries Q, and blood cells W, are used in thepresent embodiment. Image features of the eye output from the imageprocessing apparatus 10 are saved in the data server 40. Also, imagefeatures of the eye and normal value data relating to distribution ofimage features of the eye are transmitted to the image processingapparatus 10 upon request from the image processing apparatus 10. FIG. 8illustrates functional blocks of the image processing apparatus 10according to the present embodiment. This differs from the case in thefirst embodiment with regard to the point that the image processing unit130 is provided with an image feature acquiring unit 136. The imageprocessing flow according to the present embodiment is the same as thatin FIG. 5, with S510, S520, S560, S570, and S580 being the same as inthe first embodiment. Accordingly, only the processing of S530, S540,and S550 will be described in the present embodiment.

Step 530: Processing to Determine Suitability of Each Moving Image

The individual image determining unit 132 performs determinationprocessing for suitability, based on the luminance values of the framesand the inter-frame movement amount. Further, the selecting unit 134performs selection processing based on the suitability determinationresults, and forms individual images. The processing of this step willbe described in detail later with reference to the flowchart in FIG.10A.

Step 540: Processing to Determine Suitability as Image Group

Based on the individual images formed in S530, the image groupdetermining unit 133 determines suitability based on continuity of imagefeatures between images, the selecting unit 134 selects a composition ofimages with the highest suitability, and composites these to form animage. The processing of this step will be described in detail laterwith reference to the flowchart in FIG. 10B.

Step 550: Display

The display control unit 135 displays the composited image formed inS540 using the positioning parameters obtained in S520. Display ofcomposited images where capillaries such as illustrated in FIG. 9B havebeen extracted is performed in the present embodiment.

Processing to Determine Suitability for Each Moving Image

Next, the processing executed in S530 will be described in detail withreference to the flowchart illustrated in FIG. 10A. Note that S1010 andS1020 are the same as S710 and S720 in the first embodiment, sodescription thereof will be omitted.

Step 1030: Image Selection

The selecting unit 134 selects images (frames in a case of movingimages) to use for display, in increments of shooting positions, basedon the suitability determined in S1020, and forms an image. In thepresent embodiment, a high-magnification image is a moving image where acapillary region has been imaged, and an image where capillaries havebeen extracted from the moving image (hereinafter written as “capillaryimage”) is formed.

The following items (i) and (ii) can be listed as principles for formingindividual images, which is to say,

(i) maximize the number of frames used to form the capillary image(priority given to image quality), and

(ii) maximize area of capillary region (priority given to prevention ofincomplete images).

In the case of (i), all frames selected in S1020 are used to extractcapillaries. In the case of (ii), frames selected in S1020 which haveeven slight positional deviation are excluded. For example, in the caseof FIG. 6C, frames Nos. 2 through 4 are excluded. While incompleteimages do not occur, the number of frames used to extract capillaries issmaller, so the image quality tends to be lower than in the case of (i).Now, in the case of the present embodiment, frames with suitability of 1calculated in S1020 are selected, and capillary regions are extractedfollowing the principle (i), which is to say that only regions where thepixel values are positive in all selected frames are used.

Step 1040: Extracting Image Features

The image feature acquiring unit 136 detects capillaries fromhigh-magnification images Dhj, and detects the avascular area boundaryfrom the detected capillary regions. In the present embodiment, firstcapillaries are identified from the high-magnification images Dhj asblood cell component movement ranges, according to the followingprocedures, which is to say that

(a) difference processing is performed among adjacent frames ofhigh-magnification images Dhj regarding which inter-frame positions hasbeen completed (a difference moving image is generated),

(b) luminance statistics (variance) in the frame direction arecalculated at each x-y position of the difference moving image generatedin (a), and

(c) a region where luminance variance each x-y position of thedifference moving image exceeding a threshold value Tv is identified asbeing a region where blood cells have moved, i.e., a capillary region.

Note that the method for detecting capillaries is not restricted tothis; any known method may be used. For example, blood vessels may bedetecting by applying a filter that enhances linear structures to aparticular frame of the high-magnification images Dhj.

Next, the image feature acquiring unit 136 detects the avascular areaboundary from the acquired capillary regions. There is a region where noretinal blood vessels exist (avascular area) near the fovea of theretina (e.g., Dh5 in FIG. 6I). Early-stage lesions of retinal bloodvessels readily occur around the avascular area boundary, and also theavascular area spreads as lesions such as retinopathy of diabetesadvance. Accordingly, the avascular area boundary is an important objectof observation and analysis.

In the present embodiment, a circular deformable model is placed in thehigh-magnification image Dh5 situated at the center of thehigh-magnification image group, and this deformable model is deformed soas to match the avascular area boundary, thereby identifying theavascular area boundary. The position of the deformable mode regardingwhich deformation has been completed is taken as a candidate position ofthe avascular area boundary. Note that the method for identifying theavascular area boundary is not restricted to this; any known techniquemay be used.

Processing to Determine Suitability as Image Group

Next, the processing executed in S540 will be described in detail withreference to the flowchart in FIG. 10B.

Step 1050: Suitability Determination

The image group determining unit 133 calculates the following indexrelating to the image features (capillary area) acquired from thehigh-magnification images Dhj, and determines suitability of the imagegroup based on this index.(Sum of length of avascular area boundary actually acquired)/(Sum oflength of avascular area boundary candidate point sequence set in S1040)Step 1060: Image Selection and Updating Image Features

The selecting unit 134 selects images in the high-magnification imagesso that the suitability is the greatest, based on the suitabilitydetermined in S1050, and performs forming processing of the image groupbased on the selected images. Specifically, image selection is performedaccording to the following procedures, and image group formationprocessing is performed.

(3′) If there are no incomplete image features, the composited image isformed as it is and the processing ends.

(4′) If there are incomplete image features, the position of theincomplete-image-feature region is obtained.

(5′) Check whether or not there is complementary (substitute) data in aredundant region of an image including the incomplete-image-featureregion or having a side adjacent to the incomplete-image-feature region.For example, in FIG. 9A there are incomplete image features in image 6,so whether or not there is complementary data at image 6 and the leftedge of image 5 is checked.

(6′) In a case where there is complementary (substitute) data, theincomplete-image-feature region is replaced by complementary data thathas the best image quality of the complementary data (the number offrames used for capillary extraction is great), and (8′) is executed.

(7′) If there is no complementary (substitute) data, selected frames ofimages having images including incomplete-image-feature regions or aside adjacent to the region are changed so that theincomplete-image-feature region is resolved. If there are multiple frameselection methods to resolve the incomplete-image-feature region, theframe selection method where the number of selected frames is thegreatest is selected.

(8′) The number of frames ANmin′ used for generating the capillaryimage, that is the smallest number frames used for generating thecapillary image, of the capillary image group obtained in (7′), is setas the number of frames used in each capillary image. The number offrames used for capillary extraction at each shooting position ischanged to ANmin′, and the capillary image is generated again.

(9′) The capillary image generated in (8′) is used to generate acomposited image. There are no more incomplete image features, asillustrated in FIG. 6H, and a composited image where the numbers offrames used for capillary extraction are the same and are maximal isgenerated.

Note that the image features used to calculate the suitability for theimage group are not restricted to the avascular area boundary; any imagefeatures may be used. For example, in a case of determining suitabilityof an image group of four high-magnification images taken of the opticpapilla as illustrated in FIG. 9C, a cupped portion can be detected bythreshold value processing, and the suitability of the image groupdetermined based on the continuity of the boundary position of thecupped portion. Specifically, the sum of the edge of the cupped portionboundary is used as the suitability of the image group. Of course, thesuitability of the image group is not restricted to this, and may be thearea of the cupped region detected by threshold processing, for example.

The image selection method where the suitability of the image group isthe greatest is basically the same as S750 in the first embodiment. Thisdiffers from the case of the first embodiment though, in that featureextraction (cupped portion boundary detection) is performed as to theoverlaid image generated after frame selection, suitability for theimage group is determined using the continuity of the image features,and the images subjected to feature extraction are composited. Due tothis sort of suitability determination processing of the image group andimage group forming processing, the composited image with discontinuousportions in image features like in the high-magnification image Dh3 atthe lower right in FIG. 9D, can have the discontinuous portions resolvedas in FIG. 9E and the tissue to be analyzed can be analyzed undergenerally the same conditions.

According to the configuration described above, the image processingapparatus 10 determines suitability of an image group based oncontinuity of image features extracted from adjacent high-magnificationimages. Accordingly, in a case where cells and tissue to be analyzed,and lesions thereof, exist across multiple high-magnification images, acomposited image which can be analyzed under generally the sameconditions can be generated. Note that in addition to image features, atleast one condition of the relative position and luminance properties ofthe image group, which are conditions of the first embodiment, forexample, may be added as a condition for determining the suitability ofthe image group.

Third Embodiment Tomographic Image Imaging Apparatus Having AdaptiveOptic System

When compositing and displaying high-magnification adaptive optic OCTtomography images taken at different shooting positions, the imageprocessing apparatus according to a third embodiment determines thesuitability of an image group based on how small an unobservable(unanalyzable) region is when compared with a shot (analyzed) region.Specifically, description will be made regarding a case of acquiringmultiple (3×3×3=27) high-magnification images near the fovea andcompositing by positioning processing, and determining the suitabilityof the image group based on how small an unobservable (unanalyzable)region is in comparison with a region to be imaged (analyzed).

FIG. 2C illustrates the configuration of devices connected to the imageprocessing apparatus 10 according to the present embodiment. The presentembodiment differs from the first embodiment with regard to the pointthat connection is made with the tomographic image imaging apparatus 60having an adaptive optic system. The tomographic image imaging apparatus60 is an apparatus that takes tomographic images of the eye, and isconfigured as a spectral domain optical coherence tomography (SD-OCT)apparatus. The eye tomographic image imaging apparatus 60 imagesthree-dimensional images tomographic images of an eye to be examined, inresponse to operations by an operator omitted from illustration. Theimaged tomographic images are transmitted to the image processingapparatus 10.

Next, the functional blocks of the image processing apparatus 10according to the present embodiment are the same as those of the case ofthe first embodiment, so description will be omitted. The data server 40holds normal value data relating to image features of the eye anddistribution of image features of the eye, and in the present embodimentholds normal value data relating to the retina layer boundary and theshape and thickness thereof.

Next, the configuration of the tomographic image imaging apparatus 60that has an adaptive optic system will be described with reference toFIG. 11. In FIG. 11, 201 denotes a light source, for which an SLD lightsource having a wavelength of 840 nm is used in the present embodiment.A low-interference arrangement is sufficient for the light source 201,and an SLD light source having a wavelength interval of 30 nm or longeris preferably used. Also, an ultrashort pulsed laser such as aTi:sapphire laser may also be used as the light source. Light irradiatedfrom the light source 201 passes through a single-mode optic fiber 202and is guided to a fiber coupler 520. The fiber coupler 520 splits thislight to a measurement light path 521 and a reference light path 522. Afiber coupler is used which has a splitting ratio of 10:90, so that 10%of the quantity of input light goes to the measurement light path 521.The light which has passed through the measurement light path 521 isirradiated as parallel measurement light from a collimator 203. Theconfiguration downstream of the collimator 203 is the same as in thefirst embodiment, with the eye 211 being irradiate via the adaptiveoptic system and scanning optical system, and the reflected andscattered light from the eye 211 returns the same path to be guided tothe optic fiber 521 and reaches the fiber coupler 520. On the otherhand, the reference light which has passed through the reference lightpath 522 is emitted at a collimator 523, reflected at a variable opticalpath length unit 524, and returns to the fiber coupler 520 again. Themeasurement light which has reached the fiber coupler 520 is multiplexedwith the reference light and passes through optical fiber 525 to beguided to a light splitting unit 526. A tomographic image of the eye isconfigured by the control unit 217 based on interference lightinformation split by the light splitting unit 526. The control unit 217can control the variable optical path length unit 524 to acquire animage of a desired depth position. Note that by increasing theoscillation angle of the scanning optical system in the configurationillustrated in FIG. 11 and the adaptive optic control unit 216instructing not to perform aberration correction, the tomographic imageimaging apparatus 60 can operate as a normal tomographic image imagingapparatus, and can take wide-angle tomographic images (wide-angle imageD1).

Also, while the tomographic image imaging apparatus 60 having theadaptive optic system is described as being an SD-OCT in the presentembodiment, SD-OCT is not essential. For example, this may be configuredas a time-domain OCT or an SS-OCT (Swept Source Optical CoherenceTomography). In the case of SS-OCT, a light source is used where lightsof different wavelengths are generated at different times, and spectralelements to acquire spectral information become unnecessary. Also, anSS-OCT can acquire very deep images including not only the retina butalso the chorioid.

FIG. 5 illustrates the image processing flow of the image processingapparatus 10 according to the present embodiment. This the same as thecase in the first embodiment except for S510, S520, S530, S540, andS550, so only the processing of S510, S520, S530, S540, and S550 will bedescribed.

Step 510: Image Acquisition

The image acquiring unit 111 requests the image imaging apparatus 60 toacquire wide-angle images D1, high-magnification images Dhj, andcorresponding fixation target positions F1 and Fh. In the presentembodiment, the fixation target positions F1 and Fh are set at the foveaof the macula, and wide-angle images D1 and high-magnification imagesDhj are acquired. The high-magnification images Dhj are repeatedly takenNp times (Np=3 in the present embodiment) and a high-magnification imagetaken at the n'th time at the same shooting position is written as Dhj_nin the present embodiment, for example. Note that the setting method forshooting positions is not restricted to this, and may be set to anyposition.

The tomographic image imaging apparatus 60 acquires and transmits thewide-angle images D1 and high-magnification images Dhj_n, and thecorresponding fixation target positions F1 and Fh, in accordance withthe acquisition request. The image acquiring unit 111 receives thewide-angle images D1, high-magnification images Dhj_n, and fixationtarget positions F1 and Fh, from the tomographic image imaging apparatus60 via the LAN 30, and stores these in the storage unit 120. Note thatthe wide-angle image D1 and high-magnification image Dhj_n in thepresent embodiment are three-dimensional images regarding whichinter-slice positioning has already been performed.

Step 520: Positioning

The positioning unit 131 performs positioning of the wide-angle imagesD1 and the high-magnification images Dhj_n, and decides the position ofthe high-magnification images Dhj_n upon the wide-angle image D1. First,the image group determining unit 133 acquires the fixation targetposition Fh used when shooting the high-magnification image Dhj_n fromthe storage unit 120, and sets a search start point for a positioningparameter for the positioning of the wide-angle image D1 andhigh-magnification images Dhj_n, based on the relative position from thefixation target position. In a case where there is an overlapping regionamong the high-magnification images Dhj_n, the inter-image similarity iscalculated regarding this overlapping region as well, and thehigh-magnification images Dhj_n are positioned with each other at theposition where the inter-image similarity is the greatest.

Next, in a case where images of different magnification have beenacquired in S530, positioning is performed from lower magnificationimages. In the present embodiment, there are only high-magnificationimages, so positioning is performed only between the wide-angle image D1and the high-magnification images D1 j_n. Any known technique can beused for inter-image similarity or coordinate conversion techniques, andin the present embodiment, a (three-dimensional) correlation coefficientis used for inter-image similarity, and three-dimensional Affinetransform is used as the coordinate conversion technique to performpositioning.

Step 530: Processing to Determine Suitability of Each Moving Image

The individual image determining unit 132 performs determinationprocessing for suitability, based on the luminance values of the framesand the inter-frame movement amount. Further, the selecting unit 134performs selection processing based on the suitability determinationresults, and forms individual images. The determination standardacquisition and similarity determination method are the same as S710 andS720 in the first embodiment, so description will be omitted here. Next,the selecting unit 134 selects images to be used for display inincrements of shooting positions, based on the determined similarity,and forms an image. The high-magnification images in the presentembodiment are three-dimensional tomographic images such as illustratedin FIG. 12(a). Overlapping among high-magnification images is omitted inthe illustration here, to facilitate comprehension of the shootingposition.

The following can be conceived as individual image forming principles,which is to

(i) maximize the S/N ratio (priority given to image quality), and

(ii) maximize total number of pixels in three-dimensional tomographicimage (priority given to prevention of incomplete images),

in which, with regard to (i), of the tomographic images acquired at thesame shooting position in S510, the image with the highest S/N ratio isselected, and the pixel value for a region including a slice with nopixel value (image edge portion) is set to 0. The image quality is high,but incomplete images readily occur. With regard to (ii), thethree-dimensional tomographic image regarding which the processing offilling the image edges with zeros is the smallest is selected from thethree-dimensional tomographic images (a total of three) acquired inS510. While incomplete images do not occur, the S/N ratio is notnecessarily high, so the image quality tends to be lower than in thecase of (i). In the present embodiment, a slice of which the suitability1 is selected, and the luminance value in a slice of which thesuitability is −1 is a value obtained by interpolation processing bypixel values in preceding and following slices. Here, high-image qualityindividual images are formed following the principle (i), i.e., usingthe three-dimensional tomographic image of which the S/N ratio is thegreatest.Step 540: Processing to Determine Suitability as Image Group

Based on the individual images formed in S530, the image groupdetermining unit 133 determines suitability of an image group, and theselecting unit 134 selects a composition of images with the highestsuitability and composites these to form an image. In the presentembodiment, the image No. j increases in raster scanning (zigzagscanning) order from the upper left. Determination principles relatingto the composited images (image group) are as listed below in order ofpriority, which is to say

1. that no incomplete-image region is generated in the composited image,and

2. that the composited image is as high quality as possible.

Of these, 1 is a condition set for enabling within the composited imageto be observed under the same conditions. Now, in a case wheresuitability as an image group is not determined, frames need to beselected so that there are no incomplete image regions at each shootingposition for example, a composited image with lower image quality isformed in relation with condition 2 described above, as compared to acase of generating a composited image based on suitability determinationof the image group. Accordingly, performing suitability determinationtaking into consideration data continuity and complementation at theedges and overlapping portions of adjacent three-dimensional tomographicimages enables a higher-quality composited image to be obtained, whilesatisfying condition 1.

In the present embodiment, there are redundant regions among twoadjacent images as indicated by the gray regions in FIG. 12B, redundantregions among four adjacent images as indicated by the black regions inFIG. 12B, and redundant regions among eight adjacent images as indicatedby the white grid points in FIG. 12C. Specifically, the suitability ofthe image group is determined by the following procedures.

(1) Individual images generated in S530 (priority on image quality) arecomposited according to the positioning parameters obtained in S520.

(2) Whether or not there are incomplete images within the compositedimage generated in (1) is checked, and(volume (number of pixels) of composited three-dimensional image−volume(number of pixels) of incomplete images)/(volume (number of pixels) ofcomposited three-dimensional image)is calculated as the suitability of the image group.

Note that image group suitability is not restricted to this, and thatprojection images of individual images may be composited upon aprojected image of a wide-angle image based on the positioningparameters obtained in S520, and(area of composited two-dimensional image−area of incompleteimages)/(area of composited two-dimensional image)may be determined.

Based on the suitability of the image group determined above, theselecting unit 134 performs image selection so that the suitability isthe highest at each shooting position, and an image group is formedbased on the selected image. Specifically, image selection is performedaccording to the following procedures, and image group formationprocessing is performed.

(3) If there are no incomplete images, the composited image is formed asit is and the processing ends.

(4) If there are incomplete images, the position of the incomplete-imageregion is obtained.

(5) Check whether or not there is substitute data in a redundant regionof an image including the incomplete-image region or having a sideadjacent to the incomplete-image region. For example, in FIG. 12D thereare incomplete images in high-magnification images Dh3 and Dh7, sowhether or not there is complementary (substitute) data at Dh3 and thefar edge of Dh4, and Dh7 and the upper edge of Dh12, is checked.

(6) If there is complementary (substitute) data, the incomplete imageregion is replaced with complementary data that has the best imagequality (the highest S/N ratio) of the complementary data, and imagecompositing processing is performed.

(7) If there is no complementary (substitute) data, trimming processing(filling in with zeros) at the edge of the image is cancelled so thatthe incomplete-image region is the smallest, pixel values are decidedfor the remaining incomplete-image region by interpolation processingfrom nearby pixels, and then image compositing processing is performed.Thus, a composited image where there are no incomplete images and theimage quality is the highest, is formed. Note that the image groupsuitability determination principles are not restricted to theabove-described, and that any suitability may be set. For example, thelength of the avascular area boundary, which is an example of imagefeatures, may be added to the conditions for suitability besides areaand so forth.

Step 550: Display

The image group determining unit 133 the composited image formed in S540on the monitor 305. The wide-angle image D1 and high-magnificationimages Dhj_n are both three-dimensional tomographic images in thepresent embodiment, so the two following types of display are performed.

i) Projection images of the wide-angle image D1 and high-magnificationimages Dhj_n are generated with regard to the z-axial direction, and aprojected image of a high-magnification image Dhj_n is composited anddisplayed on a projected image of the wide-angle image D1.

ii) A wide-angle three-dimensional tomographic image D1″ is generated,displayed according to the pixel values of a wide-anglethree-dimensional tomographic image D1 at positions where only thewide-angle three-dimensional tomographic image D1 has been acquired, andaccording to pixels values of the high-magnification three-dimensionaltomographic image Dhj_n at positions where both the wide-anglethree-dimensional tomographic image D1 and the high-magnificationthree-dimensional tomographic image Dhj_n have been acquired. Further, aparticular scanning position on the wide-angle three-dimensionaltomographic image D1″ is indicated on the superimposed image in i) by anarrow, and a two-dimensional tomographic image of the wide-anglethree-dimensional tomographic image D1″ defined by the position of thearrow is displayed along with a superimposed image such as in i). Inthis display, not only a two-dimensional tomographic image of thewide-angle three-dimensional tomographic image D1, but also atwo-dimensional tomographic image of the high-magnificationthree-dimensional tomographic image Dhj_n is displayed superimposed.

Further, in the display of ii), the arrow indicating the displayposition of the wide-angle tomographic image D1″ can be (vertically orhorizontally) moved by the operator using the instruction acquiring unit140, so the displayed slice of the wide-angle image D1 andhigh-magnification image Dhj_n defined (displayed) along with thisoperation also changes.

Note that the method of generating the projected images is notrestricted to mean intensity projection; any projection method may beused. Also, the high-magnification images Dhj_n are not restricted tostill images, and may be moving images. Although suitability of atomographic image group has been described in the present embodiment asbeing determined based on how small an unobservable region is as to theregion of shooting, the present invention is not restricted to this. Anarrangement may be made where, the same as with the case of the secondembodiment, the image processing apparatus 10 has the image featureacquiring unit 136, and suitability of a tomographic image group isdetermined based on continuity between adjacent images, in imagefeatures extracted from the high-magnification images. For example, theimage feature acquiring unit 136 extracts layer boundaries as imagefeatures, by the following procedures. That is to say, the image featureacquiring unit 136 extracts the boundary positions of the inner limitingmembrane B1, nerve fiber layer boundary B2, inner plexiform layerboundary B4, photoreceptor inner/outer boundary B5, and retinal pigmentepithelium boundary B6, as image features from the wide-angle image D1stored in the storage unit 120, i.e., from a three-dimensionaltomographic image of the eye. The extracted image features are thenstored in the storage unit 120.

Now, feature extraction procedures from the wide-angle image D1 will bedescribed in detail. First, the extracting procedures for extractinglayer boundaries will be described. The three-dimensional tomographicimage to be processed here is conceived as being a set oftwo-dimensional tomographic images (B-scan images) and the followingprocessing is performed on the two-dimensional tomographic images.First, a two-dimensional tomographic image of interest is subjected tosmoothing processing, and noise components are removed. Next, edgecomponents are detected from the two-dimensional tomographic image, andseveral line segments are extracted as layer boundary candidates basedon the continuity thereof. Of the extracted candidates, the uppermostline segment is extracted as the inner limiting membrane B1, the linesegment second from the top is extracted as the nerve fiber layerboundary B2, and the third line segment is extracted as the innerplexiform layer boundary B4. The line segment with the largest contrastat the outer retina side from the inner limiting membrane B1 (the sizeat which the z coordinate is large in FIG. 6A) is extracted as thephotoreceptor inner/outer boundary B5. Further, the bottom line segmentof the layer boundary candidate group is extracted as the retinalpigment epithelium boundary B6. A deformable model such as Snakes orlevel sets or the like may be applied for the initial values of the linesegments, to extract more precisely. Graph cuts may be used to extractlayer boundaries. Boundary extraction using deformable models and graphcuts may be performed three-dimensionally on the three-dimensionaltomographic image, or may be performed two-dimensionally on thetwo-dimensional tomographic images. Any method may be used to extractlayer boundaries, as long as capable of extracting layer boundaries froma tomographic image of the eye.

Also, layer boundary extraction from the high-magnification images Dhj_ncan be executed based on the relative positions of the wide-angle imageD1 and high-magnification images Dhj_n, and layer boundary positionsdetected from the wide-angle image D1. That is to say, the layers aredetected from a wide-angle image D1 correlated with thehigh-magnification images Dhj_n, and the boundary of the correspondinglayers in the high-magnification images Dhj_n can be detected near theposition of the layer boundaries detected in the wide-angle image D1.

According to the configuration described above, when compositing anddisplaying high-magnification adaptive optic OCT tomographic imagestaken at different shooting positions, the image processing apparatus 10determines suitability of an image group based on how small anunobservable (unanalyzable) region is within a composited image ascompared to a region of shooting (analyzing). Accordingly, in a casewhere cells and tissue to be analyzed, and lesions thereof, exist acrossmultiple high-magnification images, a composited image which can beanalyzed under generally the same conditions can be generated.

Fourth Embodiment Comparison of Composited Image and Region Shot(Analyzed) in Different Examination

The image processing apparatus according to the a fourth embodimentdetermines suitability of an image group based on how small anunobservable (unanalyzable) region is at the time of compositing anddisplaying high-magnification adaptive optic SLO images of differentshooting positions, in a case of comparing a region shot (analyzed) in adifferent examination. Specifically, description will be made regardinga case of determining suitability of an image group based on how smallan unobservable (unanalyzable) region is in a composited image, whencompared with an image group Dhjf (f=1, 2, . . . , n−1) shot in anexamination date in the past.

The configuration of apparatuses connected with the image processingapparatus 10 according to the present embodiment is the same as in thefirst embodiment. Next, FIG. 13 illustrates functional blocks of theimage processing apparatus 10 according to the present embodiment. Thisdiffers from the case in the first embodiment with regard to the pointthat the image group determining unit 133 is provided with a temporalcomparison unit 1333. The image processing flow according to the presentembodiment is the same as that in FIG. 5, with S530, S560, S570, andS580 being the same as in the first embodiment. Accordingly, only theprocessing of S510, S520, S540, and S550 will be described. Note thatDsjf and Fsf each represent cases of acquiring SLO images and fixationtargets at different magnifications, different shooting positions, ordifferent examination dates, where s is a variable that representsmagnification, j a variable that represents shooting position No., and fa variable that represents examination date, written as s=1, 2, . . . ,smax, j=1, 2, . . . , jmax, and f=1, 2, . . . , fmax. The smaller s is,the greater the shooting magnification is (the narrower the angle ofview is). The smaller f is, the older the examination date is. Theshooting position of the lowest-magnification image (wide-angle image)is one in the present embodiment, and the shooting position No. will beabbreviated for sake of simplicity.

Step 510: Image Acquisition

The image acquiring unit 111 requests the data server 40 to transmitpast SLO images Dsjf (f=1, 2, . . . , n−1), fixation target positionsFsf, and positioning parameter values corresponding to the SLO imagesDsjf. The data server 40 transmits data corresponding to this request tothe image processing apparatus 10, and saves in the storage unit 120. Inthe present embodiment, n=4.

Next, the image acquiring unit 111 requests the SLO image imagingapparatus 20 to acquire newest examination images and fixation targetpositions, the SLO images Dsjn and fixation target positions Fsn. In thepresent embodiment, the fixation target positions F2 n and F1 n are setat the fovea of the macula, and a low-magnification SLO image D2 n andhigh-magnification images D1 jn are acquired from the SLO image imagingapparatus 20.

Step 520: Positioning

The positioning unit 131 performs positioning of the wide-angle image D2f (f=1, 2, . . . , n) and the high-magnification images D1 jf, andobtains the relative position of the high-magnification images D1 jfupon the wide-angle image D2 f, thereby generating a composited image ofthe high-magnification wide-angle images D1 jf.

In a case where there is an overlapping region among thehigh-magnification images D1 jf from the same examination date, theinter-image similarity is calculated regarding this overlapping region,and positions the high-magnification images D1 jf with each other at theposition where the inter-image similarity is the greatest. In a casewhere images of three or more different types of magnification have beenacquired, positioning is performed from lower magnification images. Forexample, in a case where an image D3 f, an image D2 kf, and an image D1jf have been acquired, first, positioning is performed between the imageD3 f and image D2 kf, and next, positioning is performed between theimage D2 kf and image D1 jf. Further, the positioning unit 131 acquiresthe fixation target position F1 f used for shooting the image D1 jf fromthe storage unit 120, and uses this to set a search start point for apositioning parameter in the positioning between the image D2 f and theimage D1 jf. Any known technique can be used for inter-image similarityor coordinate conversion techniques. In the present embodiment, acorrelation coefficient is used for inter-image similarity, and Affinetransform is used as the coordinate conversion technique to performpositioning.

Next, the wide-angle image D2 n from the newest examination and awide-angle image D2 f (f=1, 2, . . . , n−1) from a past examination arepositioned. Further, the relative position of a high-magnification imageD1 jf from a past examination as to a high-magnification image D1 jn inthe newest examination is obtained using the relative position of thewide-angle image D2 n as to the high-magnification image D1 jn therelative position of the wide-angle image D2 f as to D2 n, and therelative position of the high-magnification image D1 jf as to D2 f. Notethat positioning may be directly performed between thehigh-magnification image D1 jn in the newest examination as to thehigh-magnification image D1 jf from a past examination. The positioningunit 131 acquires fixation target positions of the images from thestorage unit 120. The positioning unit 131 sets the search start pointfor positioning of the high-magnification image D1 jn of the newestexamination and the wide-angle image D2 n of the newest examination, D2n and a wide-angle image D2 f of a past examination, and D2 f and ahigh-magnification image D1 jf of a past examination, using thesefixation target positions.

Any technique can be used for the positioning technique. In the presentembodiment, Affine transform is used for positioning first, to performgeneral positioning. Next, fine positioning is performed using the freeform deformation (FFD) technique, which is a non-rigid positioningtechnique. In either positioning, a correlation coefficient is used forinter-image similarity. Of course, any known inter-image similarity maybe used, and is not restricted to this. Thus, pixels of the newestexamination image (wide-angle image D2 n or high-magnification image D1jf) and pixels of a past examination image (wide-angle image D2 f orhigh-magnification image D1 jf) are correlated.

The present invention is not restricted to positioning based onsimilarity of pixel values. For example, the image processing apparatus10 may be provided with an image feature acquiring unit 136 in the sameway as with the second embodiment, with the image feature acquiring unit136 identifying capillary regions, and positioning then being performedbased on features using the identified blood vessel region.

Step 540: Processing to Determine Suitability as Image Group

The image group determining unit 133 determines suitability of an imagegroup based on how small an unobservable (unanalyzable) region is whencompared with a region shot (analyzed) in a different examination, theselecting unit 134 selects a composition of images with the highestsuitability, and forms a composited image. Assumption will be made inthe present embodiment that the image group is made up of nine overlaidimages such as illustrated in FIG. 6G, and that the image No. jincreases in raster scanning (zigzag scanning) order from the upperleft. Determination principles relating to the composited images (imagegroup) are as listed below in order of priority, which are

1. that no incomplete-image region is generated in the composited image,

2. that the image quality does not vary according to the shootingposition, and

3. that unobservable (unanalyzable) regions do not readily occur whencompared with a region shot (analyzed) in a different examination.

Of these, 1 and 2 are conditions set for enabling within the compositedimage to be observed under the same conditions. Now, in a case wheresuitability as an image group is not determined, for example,

(i) in a case where the composited image is generated following theprinciple of (ii) in S730 for each shooting position, there will becases where the above condition 2 is not satisfied. Also,

(ii) in a case where (i) is satisfied and also condition 2 is satisfied,the image quality is lower than a case where suitability of the imagegroup is determined and images are selected, since the number ofoverlaid images has to match that of the image with the fewest overlaidimages. Accordingly, performing suitability determination taking intoconsideration data continuity and complementation at the edges andoverlapping portions of adjacent images enables a higher-qualitycomposited image to be obtained (overlaying being performed with agreater number of images), while satisfying conditions 1 through 3.

In the present embodiment, there are redundant regions between twoadjacent images, such as indicated by the gray regions in FIG. 6F, andredundant regions between four adjacent images, such as indicated by theblack regions in FIG. 6F. Specifically, the suitability of the imagegroup is determined by the following procedures.

(1) Individual images generated in S530 (priority on image quality) arecomposited according to the positioning parameters obtained in S520.

(2) The temporal comparison unit 1333 checks whether or not there areincomplete image regions in a case of comparing the composited imagegenerated in (1) with composited images generated in past examinations.

Specifically, a logical disjunction (∩_(f)(∪_(j)D1 jf) is a region ofcomparison as to the compositing of the image group D1 jf of eachexamination, i.e., as to the logical disjunction (∪D1 jf) of D1 jf.There are three image groups serving as objects of comparison (differentexaminations) in the present embodiment, so whether or not there areincomplete-image regions is checked in a case where the region ofcomparison is a logical disjunction (∪D1 j 1)∩(∪_(j)D1 j 2)∩(∪_(j)D1 j3), as to the compositing of the image group D1 jf of each examination,i.e., as to ∪D1 jf.

Accordingly,(area of composited image generated in (1)−area of incompleteimages)/(area of composited image generated in (1))is calculated as the suitability of the image group. Note that themethod for setting a region for comparison is not restricted to this;any comparison region may be set. A region for comparison may also beset manually. The selecting unit 134 performs image selection in thehigh-magnification images so that the suitability is the highest, basedon the suitability of the image group, and performs forming processingof an image group based on the selected images. Specifically, imageselection is performed according to the following procedures, and imagegroup formation processing is performed.

(3) If there are no incomplete images, the composited image is formed asit is and the processing ends.

(4) If there are incomplete images, the position of the incomplete-imageregion is obtained.

(5) Check whether or not there is complementary (substitute) data in aredundant region of an image including the incomplete-image region orhaving a side adjacent to the incomplete-image region. For example, inFIG. 6G there are incomplete images in image 6 and image 9, so whetheror not there is complementary data at image 6 and the left edge of image5, and in image 9, is checked.

(6) If there is complementary (substitute) data, the incomplete imageregion is replaced with complementary data that has the best imagequality (the number of overlaid images is the greatest) of thecomplementary data, and (8) is executed.

(7) If there is no complementary (substitute) data, selected frames ofimages having images including incomplete-image regions or a sideadjacent to the region are changed so that the incomplete-image regionis resolved. If there are multiple frame selection methods to resolvethe incomplete-image region, the frame selection method where the numberof overlaid images is the greatest is selected.

(8) The number of overlaid images ANmin that is the smallest number ofoverlaid images out of the overlaid image group obtained in (7) is setas the number of overlaid images of the composited image, the number ofoverlaid images at each shooting position is changed to ANmin, and theoverlaid image is generated again.

(9) The overlaid image generated in (8) is used to generate a compositedimage. There are no more incomplete images, as illustrated in FIG. 6H,and a composited image where the numbers of overlaid images are the sameand are maximal is generated. Note that the forming is performed in thepresent embodiment based on determination of suitability of the imagegroup.

While the composited image formed based on the determination ofsuitability as an image group has been described as being a still image(overlaid image) in the present embodiment, the present invention is notrestricted to this. For example, suitability determination may beperformed taking into consideration complementation of data at the edgesand overlapping portions of adjacent moving images, and a moving imagemay be composited and displayed thereupon, as illustrated in FIG. 6J.The flow of the basic processing in a case of composited display ofmoving images is the same as in a case of composited display of stillimages, but the following points differ.

(i) A time phase data acquiring apparatus 50 such as illustrated in FIG.2B is connected to the image processing apparatus 10, and time phasedata is acquired simultaneously with the moving image. Time phase datais biosignal data acquired by a sphygmograph, for example. Referencingthe time phase data yields the playback cycle of each moving image. Theplayback cycle is aligned among the moving images by frame interpolationprocessing of the moving images (between shooting positions, or betweenexaminations, or both).

(ii) The longest continuous frame section from which frames withabnormal luminesce have been removed is selected in the image formationprocessing in increments of shooting position.

(iii) Suitability determination is performed in the suitabilitydetermination processing for the image group according to the followingprinciple, which is to composite and display moving images with as manyframes (pulse cycles) as possible, where the number of playback frames(pulse cycles), in which the number of playback frames (pulse cycles)with no incomplete-image regions occurring in the composited image isgenerally the same for each shooting position, is generally the samebetween examinations.

(iv) The frames selected in (6) (7) (8) of the image formationprocessing as an image group are set as a continuous frame section.

Accordingly, incomplete images are eliminated from the compositeddisplay of the moving image, and a composited moving image with thelongest continuation of frames having the same number of playback framesis formed. In a case where no time phase data is acquired, a compositeddisplay may be made as a moving image without adjusting the playbackclock time. Although the above conditions 1 through 3 have been used assuitability for the image group (still image group and moving imagegroup) in the present embodiment, the present invention is notrestricted to this; any suitability may be set.

Step 550: Display

The display control unit 135 uses the positioning parameters obtained inS520 and the image group formed in S540 to composite and display thehigh-magnification images D1 jn upon the wide-angle image D2 n.According to the configuration described above, when compositing anddisplaying high-magnification adaptive optic SLO images taken atdifferent shooting positions, the image processing apparatus 10determines suitability of an image group based on how small anunobservable (unanalyzable) region is, in a case of comparing a regionshot (analyzed) in a different examination. Accordingly, in a case wherecells and tissue to be observed or analyzed, and lesions thereof, existacross multiple high-magnification images, a composited image which canbe observed or analyzed under generally the same conditions can begenerated.

Other Embodiments

Although a case of shooting one image group and determining suitabilityhas been described in the above embodiments, embodiments of the presentinvention are not restricted to this. That is to say, in a case wheremultiple image groups are shot in one examination as illustrated inFIGS. 14A and 14B, images in the image groups may be tentativelyselected in the same way as in the above embodiments, after whichsuitability is determined among the image groups and images are selectedbased on the suitability among the image groups.

For example, in a case of acquiring adaptive optic SLO images ofdifferent magnifications at each shooting position and generating acomposited image as illustrated in FIG. 14A, suitability for the imagegroup is determined for each magnification, and images are tentativelyselected. Thereafter, determination of suitability may be made among themagnifications (area of logical disjunction of high-magnification imageD1 j group that does not extend outside of shooting region oflow-magnification image D2 k group/area of logical disjunction ofhigh-magnification image D1 j group), and images may be selected basedon the suitability among the image groups.

Also, in a case of having multiple image groups MnDhj (multi-placementtype) as illustrated in FIG. 14B, first, suitability is determinedwithin each image group as in the above-described embodiments, andimages are tentatively selected. Thereafter, suitability among the imagegroups is determined based on suitability relating to the luminanceproperties (e.g., average luminesce or S/N ratio similarity within theimage group) among the image groups (e.g., among adjacent image groups),and images may be selected based on the suitability among the imagegroups. In a multi-placement type, the image groups may be away fromeach other, may be adjacent, or may be overlapping. Cases where the sizeof the image group (e.g., number of acquired positions of images makingup the image group) differs among image groups are also included in thepresent invention.

Also, in a case of acquiring image groups of a multi-placement type inmultiple magnifications as illustrated in FIG. 14C, either case of firstobtaining suitability among image groups with different magnificationsand suitability among image groups with different acquisition positionsis included in the present invention.

Also, while the images to be positioned in the above describedembodiments have been realized as SLO images or tomographic images ofthe eye, the embodiments of the present invention are not restricted tothese. For example, the wide-angle image D1 may be realized as a funduscamera image and the high-magnification image Dh realized as an adaptiveoptic fundus camera image. Also, the images may be realized as imageswith different modalities, such as the wide-angle image D1 being awide-angle SLO image and the high-magnification image Dh being aprojected image of an adaptive optic system tomography image, or thewide-angle image D1 being an adaptive optic system tomography image andthe high-magnification image being an adaptive optic SLO image. Thisalso may be realized as a configuration where a complex machine of theadaptive optic SLO image imaging apparatus 20 and the tomographic imageimaging apparatus 60 is directly connected to the image processingapparatus 10.

Further, while the present invention has been realized as an imageprocessing apparatus in the above-described embodiments, embodiments ofthe present invention are not restricted just to an image processingapparatus. For example, the present invention may be realized assoftware executed by a CPU of a computer. It is needless to say that astorage medium storing this software also makes up the presentinvention.

When selecting images from moving images, at least one of the edges andoverlapping portions of multiple images is preferably used. Preferablyfurther provided is a comparing unit to compare a composited image and awide-angle image that is of a wider angle of view than each of themultiple moving images (e.g., the image group determining unit 133 inFIG. 1). Thus, a determining unit can determine values indicatingcontinuity, based on comparison results by the comparing unit.Preferably further provided is a display control unit (e.g., the displaycontrol unit 135 in FIG. 1) for displaying the composited image wherethe selected images have been composited, on a display unit. Also, bloodvessels are preferably automatically (or instructed manually by the useror semi-automatically) extracted from the composited image, with themovement speed and so forth of blood cells in the extracted bloodvessels being measured. Note that the multiple images do not have to becomposited; each of multiple images may be displayed arrayed on thedisplay unit, or may be switched one at a time and displayed on thedisplay unit.

The present invention is also realized by executing the followingprocessing. That is to say, processing where software (program) thatrealizes functions of the embodiments described above are supplied to asystem or apparatus via network or various types of storage media, and acomputer (or CPU or microprocessor unit (MPU) or the like) of the systemor apparatus reading out and executing the program.

Embodiments of the present invention can also be realized by a computerof a system or apparatus that reads out and executes computer executableinstructions recorded on a storage medium (e.g., non-transitorycomputer-readable storage medium) to perform the functions of one ormore of the above-described embodiment(s) of the present invention, andby a method performed by the computer of the system or apparatus by, forexample, reading out and executing the computer executable instructionsfrom the storage medium to perform the functions of one or more of theabove-described embodiment(s). The computer may comprise one or more ofa central processing unit (CPU), micro processing unit (MPU), or othercircuitry, and may include a network of separate computers or separatecomputer processors. The computer executable instructions may beprovided to the computer, for example, from a network or the storagemedium. The storage medium may include, for example, one or more of ahard disk, a random-access memory (RAM), a read only memory (ROM), astorage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Application No.2014-181342, filed Sep. 5, 2014, which is hereby incorporated byreference herein in its entirety.

What is claimed is:
 1. An image processing apparatus that generates oneimage by using at least one frame each of a plurality of moving imagesobtained by taking moving images of a plurality of different regions ofan eye at different times, the apparatus comprising: a deciding unitconfigured to decide the at least one frame in each of the plurality ofmoving images, so that regions which have actually been shot areincluded in the plurality of moving images in the plurality of regions;and an image generating unit configured to generate one image by usingthe at least one frames decided from each of the plurality of movingimages.
 2. The image processing apparatus according to claim 1, whereinthe deciding unit decides the at least one frame, so that similarityamong the at least one frames in the plurality of moving images is equalto a threshold value or higher.
 3. The image processing apparatusaccording to claim 1, further comprising: an image acquiring unitcommunicably connected to an ophthalmologic apparatus, to acquire theplurality of moving images obtained by the ophthalmologic apparatustaking the moving images of the eye.
 4. The image processing apparatusaccording to claim 3, the ophthalmologic apparatus further including ascanning optical system configured to scan measurement light on the eye;wherein the image acquiring unit controls the scanning optical system sothat the measurement light is repeatedly scanned on the plurality ofregions of the eye, thereby acquiring the plurality of moving images. 5.The image processing apparatus according to claim 3, the ophthalmologicapparatus further including a scanning optical system configured to scanmeasurement light on the eye; wherein the image acquiring unit controlsa position of a fixation target to fix the eye so that the measurementlight is repeatedly scanned on each of the plurality of regions of theeye, thereby acquiring the plurality of moving images.
 6. The imageprocessing apparatus according to claim 3, the ophthalmologic apparatusfurther including a wavefront correction device configured to correct awavefront of at least one light of measurement light and returning lightfrom the eye, wherein the image acquiring unit acquires the plurality ofmoving images obtained by taking moving images of the eye using light ofwhich the wavefront has been corrected.
 7. An image processing apparatuscomprising: an image acquiring unit configured to acquire a plurality ofmoving images shot at different positions of an eye; a determining unitconfigured to determine a value indicating continuity of properties of aplurality of images made up of images obtained by being selected fromeach of the plurality of acquired moving images; a selecting unitconfigured to select images from the plurality of acquired movingimages, so that the determined value satisfies a predeterminedcondition; and a display control unit configured to display an imagegenerated from the selected images, on a display unit.
 8. The imageprocessing apparatus according to claim 7, wherein the determining unitdetermines a value indicating continuity of at least one of relativeposition, luminance properties, and image features of the plurality ofimages.
 9. The image processing apparatus according to claim 7, whereinthe determining unit determines a value indicating continuity ofproperties in the plurality of images, in at least one of edges andoverlapping regions of the plurality of images.
 10. The image processingapparatus according to claim 7, wherein the determining unit determinesa value indicating the continuity, based on an image generated from theplurality of images.
 11. The image processing apparatus according toclaim 7, wherein the determining unit determines a value indicating thecontinuity, based on at least one of an area of an image generated fromthe plurality of images, and a length of an avascular area boundary. 12.The image processing apparatus according to claim 7, further comprising:a comparing unit configured to compare an image generated from theplurality of images, with a wide-angle image which has a wider angle ofview than each of the acquired plurality of moving images, wherein thedetermining unit determines a value indicating the continuity, based oncomparison results from the comparing unit.
 13. An image processingapparatus comprising: an image acquiring unit configured to acquire aplurality of images relating to an eye at different acquiring positions;a determining unit configured to determine suitability of an image groupof the plurality of acquired images; and a selecting unit configured toselect at least one of a region or frame within the images, or images atgenerally the same acquiring position, based on the determinationresults from the determining unit.
 14. The image processing apparatusaccording to claim 13, wherein the determining unit determines thesuitability of the image group based on at least one of relativeposition between images of a same examination, continuity of luminanceproperties, similarity of image quality, and image features of theplurality of acquired images.
 15. The image processing apparatusaccording to claim 13, wherein the determining unit determines thesuitability based on at least one of relative position among imagegroups from different examinations, continuity of luminance properties,and similarity of image quality.
 16. The image processing apparatusaccording to claim 13, further comprising: a display control unitconfigured to display one image generated by using regions or frames ofimages selected by the selecting unit, on a display unit.
 17. The imageprocessing apparatus according to claim 13, further comprising: ameasuring unit configured to measure movement speed of blood cells inregions or frames of images selected by the selecting unit.